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ABSTRACT 

The relationship between cloze and multiple-choice 
tests as measures of reading comprehension at two grade levels, fifth 
and eighth, was investigated. The reading passages used in the 
testing were at difficulty levels either 2 years below, 2 years 
above, or at the subjects* grade level. The final subject sampling 
included 708 students from each grade. Each of the 35 fifth-grade 
classes and 32 eighth-qrade classes were randomly assigned to either 
cloze or multiple-choice tests. The California Achievement Test was 
given to all subjects before the study began. Independent variables 
considered were the type of test, the grade level, and the passage 
difficulty. An overall analysis of results showed the cloze test to 
be sensitive to grade levels but less sensitive to change in passage 
difficulty. The two tests overlapped in measuring some parts of 
comprehension, but the multiple-choice tests measured more. It was 
concluded that caution should be used when applying cloze test scores 
as measures of comprehension until detailed analyses show what the 
cloze procedure is measuring and the grade and difficulty ranges for 
which it is appropriate. References are included. (NH) 
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THE RELATION BETWEEN CLOZE AND MULTHE -CHOICE TEST SCORES 
AS A FUNCTION OF RELATIVE PARAGRAPH DIFFICULTY AND GRADE LEVEL 

Ludwig Mosberg, Thomas CPottor and Robort K.Cornoll 



INTRODUCTION 

* 

During the last fifteen years, cloze procedures of varying form 
have been used to assess stimulus characteristics of printed materials 
with adults and children. The cloze test form frequently used in these 
studies is constructed by removing every fifth or every tenth word from 
a passage and replacing it by a blank of a standard length. The subject's 
task is to determine the words which were removed and write them in the 
appropriate blank. His score is the number of words he replaces which 
are identical to those deleted. The resulting scores have been considered 
as indices of passage readability by such authors as Taylor (1953, 1956, 
1957) and Bormuth (1962). The validity of this contention is based on 
the finding that cloze scores rank passages of varying difficulty in 
the same order as traditional "readability formulas" such as the Dale- 
Chall (1948). 

Other authors have suggested that cloze scores also provide a 
measure of a subject's comprehension of printed material. In 1959, 

Rankin suggested that cloze tests be considered a measure of reading 
comprehension. Jenkinson (1957) reported correlations as high as .82 
between cloze test results and objective questions based on the same 
material with adult subjects. 

Recent studies have indicated that cloze test scores are positively 
related to standardized, reading comprehension test scores for elementary 
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and junior high school subjects. Gallant (1965) reports correlations 
of .65 to .81 for primary grade subjects. Ruddell (1965) reports corre- 
lations of .61 to .72 using fourth grade subjects. Positive relationships 
have also been reported between cloze test scores and multiple-choice 
test scores over the same passages. Bormuth (1962, 1967) reports correlations 
ranging from .45 to .92 using fourth, fifth, and sixth grade subjects. 

These studies have used a restricted sample of reading material, 
characterized by an unspecified or narrow range of passage difficulty, 
and a relatively small number of ^s. Furthermore, different procedures 
for constructing cloze tests, different scoring procedures and different 
criterion measures have been used, making comparisons between studies 
difficult. 

The present study investigated the relationship between cloze and 
multiple-choice tests at two grade levels (fifth and eighth), with 
reading passages at difficulty levels either two years below, two years 
above or at ^'s grade level. At each grade level and passage difficulty 
level a relatively large number of reading passages were tested over a 
sizable subject sample. The multiple-choice tests were taken from a set 
of published reading exercises and the cloze passages were identical to 
those used in the multiple -choice tests. To avoid order effects, Ss 
were matched on the basis of standardized reading achievement test scores: 
one member of each pair was assigned at random to receive the multiple- 
choice test while the other member received the identical passage in 
cloze form. 

METHOD 

MATERIALS 

All reading materials were taken from the SRA Rate Builders, which 
are part of the SRA Reading Laboratory Kits (Parker, 1957, 1958, 1959, 

1960, 1963, 1964). These Rate Builders consist predominantly of concise 
non-fiction selections covering a wide range of topics including physical, 
biological and social sciences, history, human interest, etc. Following 
each selection were from four to ten multiple-choice questions, the number 
depending upon the length of the selection and its difficulty. Each 
selection was assigned a reading difficulty level by SRA based on the SRA 
Reading formula, sometimes in conjunction with other reading formulae 
such as the Spache (1953) or Lorge (1944) formulae. Initially 90 passages 
were selected at random, 15 at each of six difficulty levels, third, fifth, 
sixth, seventh, eighth, and tenth grade. The length of the passages varied 
from 75 words for third grade passages to 200 words for tenth grade passages. 
To make the number of response units more equivalent for all subjects, 
two third grade passages were administered to bring the total number of 
words read to 150. The mean number of words for the fifth, sixth, seventh, 
eighth and tenth grade selections were 150, 165, 180, 180, 200 respectively. 

To maintain a constant ratio between cloze deletion and multiple-choice 
questions as well as to increase the reliability of the multiple-choice 
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test, additional questions were added to the SRA questions. A mean ratio 
of three multiple-choice items to eight cloze deletions was maintained. 
Setting the mean number of questions for the tenth grade passages (200 
words) at 15, the mean number of questions for third, fifth, sixth, 
seventh, eighth grade passages were 10.5, 10.5, 12.4, 13.5 and 13.5 
respectively. This meant that between three and five questions were 
added to each selection. The additional questions were similar in form 
and style to the SRA questions and used vocabulary which appeared in the 
text. 



The cloze tests were constructed from the same selections used in 
the multiple -choice tests. A fifteen space blank was substituted for 
every fifth word. At each difficulty level, 1/5 of the passages began 
deletions with the first, second, third, fourth, or fifth word of the 
selection. Passages were randomly assigned to initial deletion conditions. 

A pilot study using 100 fifth graders and 100 eighth graders indicated 
that when the SRA formula was used to determine passage difficulty, there 
was no difference in fifth graders 1 performance on the third or fifth 
grade passages on either the cloze or multiple- choice tests. Similarly, 
eighth graders performed equally well on sixth and eighth grade passages. 

To shed light on the lack of differential performance on third vs. fifth 
and sixth vs. eighth grade passages, the reading difficulty of the passages 
were re-computed using the Dale-Chall (1948) formula. The Spache (1953) 
formula was applied to third grade passages because the Dale-Chall formula 
is known to be insensitive to difficulty of selections appropriate to 
the early elementary grades. When passage difficulty was thus re-computed, 
there was no difference between third and fifth grade or sixth and eighth 
grade passages selected from the SRA materials. The SRA grade placements 
were generally lower on the fifth and eighth grade material than the Dale- 
Chall assignments. The SRA formula takes into account the number of 
syllables per word and sentence length whereas the Dale-Chall and Spache 
formulae use sentence length and vocabulary difficulty based on frequency 
counts. Therefore, the Dale-Chall and Spache formulae were used to 
determine the difficulty level oi> the SRA passages used in the main study. 
Reading difficulty grade placements were re-computed for all SRA Rate 
Builders and a new sample of paragraphs (15 at each of the six grade 
levels) was chosen randomly for use in the final study. 

The pilot study data revealed that the questions added by E were 
consistently about ten percent more difficult than the SRA questions. 

It was determined that this difference would not materially change the 
results of the experiment and the practice of adding questions was again 
used in the main study. 

PROCEDURE 

Each of the 35 fifth grade classes and 32 eighth grade classes was 
randomly assigned to either the cloze condition or the multiple-choice 
test condition. Prior to testing, California Achievement Test (CAT) 
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reading scores were obtained on all JSs. The CAT was administered approxi- 
mately six months prior to this study. On the basis of the CAT reading 
scores Ss within each grade level were matched using two criteria: (a) 
that one S in each pair be in a class assigned to the cloze condition 
and the other S in a class assigned to the multiple-choice condition; 

(b) that there be no more than a two month difference in CAT grade 
equivalence scores for the two Ss. Over 95 percent of the subjects were 
matched on the basis of identical scores. All E5s for whom a matched jS 
could not be found were eliminated from the study. This resulted in ~~ 
eliminating approximately five percent of the initial sample. 

Three independent variables were investigated: (a) Type of test 

(Cloze or Mu It ip le- Choice) ; (b) Grade level (fifth or eighth); (c) 

Passage difficulty (two years above or below grade level or at grade 
level). The design of the experiment, therefore, was a 2 x 2 x 3 fixed 
effects factorial design, yielding a total of 12 conditions with 118 Ss 
per group. Within each grade level, matched pairs were assigned randomly 
to one of the three passage difficulty levels. Within difficulty levels 
each matched pair was assigned randomly to one of the 15 selections at 
that level. The original design called for 10 matched pairs to receive 
the same selection. However, due to absenteeism, the final sample consisted 
of between seven and ten subject pairs receiving each passage. 

Testing was conducted by four Es, each of whom held an elementary 
school credential; each was shown how to administer the tests. For both 
the cloze and multiple-choice tests, the instructions consisted of an 
explanation of the task and a practice sample which the E and Ss negotiated 
together (see appendix). For both tests Ss were given 25 minuses to complete 
the test. The pilot study revealed that 25 minutes was sufficient time 
for over 95 percent of the Ss to complete either test. 

SUBJECTS 

The initial sample consisted of 1038 fifth graders and 984 eighth 
graders in a Southern California school district. This district is 
a predominantly lower-middle to upper-middle class community. Due to 
transfers, absenteeism and inability to match £s, the final sample 
consisted of 708 fifth graders and 708 eighth graders. 

RESULTS 

Scores on both cloze and multiple -choice (MC) tests were converted 
to proportion of correct responses in order to make appropriate comparisons. 
To ensure that the random assignment of Ss to the three levels of passage 
difficulty did not produce significant differences between groups on 
reading ability, an analysis of variance was performed for CAT scores 
for the three difficulty- level samples at each grade level. No reliable 
differences were obtained. Means art presented in Table 1. Groups 
within the fifth and within the eighth grades appeared comparable iti 



Table 1 



Comparison of CAT scores 


as a Function of Grade and Passage Difficulty 


Difficulty Level 


Fifth Grade 


Eighth Grade 


High 


4.731 (1.302)* 


8.136 (1.859) 


Medium 


4.856 (1.271) 


8.095 (1.933) 


Low 


4.739 (1.506) 


8.186 (1.940) 


Mean 


4.775 (1.361) 


8.186 (1.908) 



* Numbers In parentheses are standard deviations. 



terms of reading ability. The mean CAT score for the fifth grail ?s was 
4.775 and 8.186 for the eighth graders. 

Separate 1-way analyses of variance comparing the SRA MC items with 
those written by the Es at each grade level and passage difficulty level 
indicated that in all cases performance on the SRA items was reliably 
higher than on our own items. This difference of approximately 12 to 
15 percent was consistent over groups except for the high difficulty 
passages for both the fifth and eighth grades, where differences were 
30 and 7 percent respectively. The 30 percent difference between scores 
on the SRA items and our items may reflect an over-estimation on the Es 1 
part of the abilities of Ss at this intermediate grade level. The 7 
percent difference for the tenth grade material may reflect a greater 
facility on the Es' part to construct more appropriate questions as the 
reading ability of the Ss nears that of the test constructors. The 
difference in correlation between cloze tests and SRA items and between 
cloze and our own items was .065. Since the only primary effect of our 
questions was to reduce the absolute value of the proportion correct, the 
two sets of items were pooled for all analyses. This procedure had the 
advantage of reducing ceiling effects on the MC tests and providing more 
reliable scores.. 

Table 2 shows the mean proportion correct and the standard deviations 
for each of the 12 cells of the design. As expected, the mean proportion 
correct on the MC tests was superior to those on the cloze tests. Second, 
as difficulty level of the passages increased, scores on both cloze and 
MC tests decreased. Third, eighth graders showed consistently higher 
scores than fifth graders on both tests and at all levels of passage 
difficulty. A 3x2x2 analysis of variance for independent groups 
(Table 3) indicated that all main effects were statistically reliable 
at the .01 level of significance; the Type of Test x Passage Difficulty 
Level interaction also was statistically significant at the .01 level. 

This interaction effect is graphically presented in Figure 1. It is 
evident that the interaction effect is due to the lack of difference 
between cloze performance at the lowest and medium difficulty levels at 
each grade level (.321 vs. .318; .405 vs. .394). These results, in 

combination with the fact that there is a linear decrease in performance 
over difficulty levels on the MC tests, suggest a lack of sensitivity of 
the cloze procedure at the lower end of the difficulty dimension. 

It should be noted that the overall analysis of variance treated 
the cloze and MC groups as independent groups even though matched Ss 
(CAT scores) were assigned to the two test conditions. The correlation 
between cloze and MC scores for matched Ss (.489) was such that it was 
felt that treating this variable as a repeated measure was inappropriate. 
Treating the two test conditions as independent groups is a somewhat 
conservative test, in the sense that employing a repeated measures design 
would have decreased the error term and thereby increased the probability 
of obtaining significance. However, inspection of the data (Figures 2 
and 3) revealed that, had a repeated measures design been used, none of 
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Table 3 

Analysis of Variance of Proportion of Correct Responses 



Source 


df 


SS 


MS 


P 


Type of Test (T) 


1 


42.772 


42. 772 


1326.275** 


Grade (G) 


1 


2. 979 


2.979 


92.386** 


Passage Difficulty (D) 


2 


5.298 


2.649 


82.145** 


T x G 


1 


0.020 


0.020 


0.618 


T x D 


2 


0.780 


.0.390 


12.100** 


G x D 


2 


0.126 


0.063 


14951 


T x G x D 


2 


0.796 


0.398 


1.234 


Error 


1404 


45.281 


0.032 


• • * * # 

' v * * 










— S 



* p<.05 

** p <.01 
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Figure 1. Proportion Correct ee e Function o£ 
Type of Test end Pessege Difficulty 
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the non- significant Interactions would have approached statistical 
significance. 

A series of Pearson product moment correlation coefficients were 
computed for cloze and MC scores and for CAT and cloze and GAT and MC 
scores (Tables 4 and 5). The correlations were consistently higher for 
fifth graders than eighth graders. These correlations also show that 
for the fifth graders the highest correlation Is obtained for the easiest 
material whereas for eighth graders the correlation Is lowest for the 
easiest material. Figure 4 shows linear least-squares best-fits for each 
of the six experimental conditions, where the MC test score Is the criterion 
measure. Figure 4 shows group 5-L (fifth graders taking the 
easiest passages) and 8-M (eighth graders taking the medium difficult 
passages) excepted, that the slopes are parallel within grade levels and are 
somewhat steeper for the fifth than for the eighth grade. 

The correlations between CAT and cloze scores are consistently 
higher than those correlations between CAT and MC scores (Table 5). This 
Is somewhat surprising since the CAT Is primarily a multiple- choice test. 

A partial explanation of this effect was found In scatterplots of the 
data, which showed a moderate but noticeable celling effect on the MC 
tests. The correlations are consistently higher for the fifth graders 
than for the eighth graders and for the fifth graders the highest corre- 
lations are obtained for the easiest material, whereas the highest 
correlations for the eighth graders are obtained for the most difficult 
material. 



DISCUSSION 

The results of the overall analysis of variance Indicate that the 
cloze test Is sensitive to grade level but Is less sensitive to changes 
In passage difficulty, at least at the lower end of the difficulty 
dimension. The correlations suggest that the cloze procedure does, In 
fact, measure some component of comprehension as measured by MC tests, but 
it Is apparent that a large component of comprehension as measured by MC 
tests Is not accounted for by the cloze procedure. However, it should be 
noted that the correlations between MC and cloze test scores were calculated 
on the basis of matched Ss. Insofar as the matching was not perfect, the 
correlations between the two types of tests should be depressed. One 
might hypothesize that the true correlations are somewhere between those 
obtained In this study and those reported by Bormuth (.92) (1967). (Bormuth 
used the same subjects and material for both tests, which Introduced 
possible order-effects, and more restricted subject and passage samples.) 

Although eighth graders performed better than fifth graders on both 
MC and cloze tests, the correlations at all passage difficulty levels 
wire lower for eighth graders than for fifth graders. The superior 
performance of eighth graders was most likely a function of their superior 
ability to follow Instructions (both grades received the same instructions) 
and greater experience in test taking. These same variables may also 
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Figure 2. Proportion Correct «• a Function of 
Typo of Toot and Grade Laval 
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PROPORTION CORRECT 
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Figure 3. Proportion Correct et e Function 
of Grede end Peseege Difficulty 









Table 4 



Correlations between Close and Multiple-Choice Scores 


* * 

• / 

Difficulty' Level 

i * • i 1 


* . i . * * * . * 

« • • * % 

Fifth Grade 

f f * t # « * • • ' 


Eighth Grade 


Low 


.649 

> 


.190 


) t 

* 

Medium 


.424 


.367 


. i • 

High 

i 


.434 


.267 


Overall 


< 

.535 


.335 
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T«bl« 5 

Correlations between CAT and Multiple-Choice Scoraa 
and batwaan CAT and Close Scoraa 



Dl££lculty Level 


Multiple-Choice 




Close 




Fifth Grade 


Eighth Grade 


Fifth Grade 


Eighth Grade 


Low 


.691 


.461 


.694 


.510 


Medium 


.619 


.478 


.646 


.510 


High 


.560 


.554 


.677 


.601 



Overall 



.484 



.561 



Figure 4. Best Fit Lines and Regression Coefficients 
for Each of the Six Experimental Conditions 
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Regression Coefficients 

5-L .56405 
5-M .24751 
5-H .26915 
8-L .15380 
8-M .27498 




0 .200 .400 .600 .800 1.00 

MULTIPLE-CHOICE SCORES 

5«Fifth grade; 8»Eighth grade; L*Low Passage Difficulty; Medium 
Passage Difficulty; H*High Passage Difficulty. 
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be responsible for the higher correlations for the fifth graders than 
eighth graders. Inspection of the scatterplots Indicate greater 
ceiling effects on the MC test for the eighth graders than fifth graders 
which have the effect of reducing the correlations. The additional 
finding that the highest correlation for the fifth graders la obtained 
with the easiest material while the eighth graders show the lowest 
correlation with the easiest material can be similarly explained. For 
eighth graders, as passage difficulty increased ceiling effects decreased; 
hence, the correlation should be depressed for the easiest material. 

It may be concluded that one should proceed cautiously when using 
cloze test scores as measures of comprehension (or predictors of MC test 
score magnitudes) until a more detailed analysis of what the cloze procedure 
is measuring and of grade and difficulty ranges at which the cloze 
procedure is an appropriate measure have been established. 
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CLOZE TEST DIRECTIONS 

Good morning, my name is . 

Your teacher may have told you that I am going to give you a teat. 

It is important that you do the beat job you can, but, the scores will 
not go on your school records. 

Please clear your desks. 

I need four volunteers, (directions to volunteers) 

Two of them pass out the tests to the person named so that the 
name is up. 

Two of them hand out the pencils. 

Please do not turn the booklet over until I ask you to do so. 

If your pencil breaks during the teat, raise your hand and I'll give 
you another. 

Please leave the pencil on the desk until we start the test. 

(When all of the materials have been handed out, SAY ) 

Now turn over the test booklet . Find the word DIRECTIONS 

Read the directions silently while I read them aloud. They say 

This is a new kind of a test. You will read some short paragraphs. 

Words have been left out of the paragraphs and blank spaces were put 
where the words were removed. 

Your -lob will be to decide what word was left out of each space and write 
that word in the snace . Reading the other words in the sentence will help 
you decide what the missing word should be. 

It will help you in taking the test if you will remember these things: 

* 

1. Write only one word in each blank. 

2. Fill every blank. Don't be afraid to guess. 
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3. Skip hard blanks and coma back to them later. 

4. Wrong spelling will not count against you if 
we can tell what word you meant. 

5. Most of the blanks will be filled with ordinary 
words, but a few will be 

numbers like 3.427 or $12 or 1954 

contractions like. ..... can't or weren't 
abbreviations like ..... Mrs , or U.S.A. 

6. Write neatly. 

Let's try a couple of samples to see how it words. 

Look at Sample (A) Birthday. 

A word is missing from the sentence. What do you think the missing 
word is? #(Happy) Write the word in the blank. 

What was the clue word? #(Blrthday) 

!, ? ow. . .Look at Sample (B) are you going? 

What do you think the missing word is? #(Where) Write it in the blank. 
What clues did you use? #(questlon mark - going) 

Now look at Sample (C) (The Beaver) 

Indians call beavers the "little men of the woods." But 

they aren't really so very little. Most 

beavers grow to be three or four feet long <xnd 

weigh from 30 to 40 pounds. These "little men 

o/ the woods" are busy most of the 

time. That ia why we say, "He's bu8y 

as a beaver." 

Sometimes a paragraph like this one has clues that will help you 
fill in the blanks at the beginning that were hard for you. If 
you have trouble with a blank skip it and come back to it later. 
Remember! Put only one word in each blank. 

Xsnq (e sj (g 

38 om (i jo (9 o* (fi (* ©aaqa (£ asom ( z a guess (i : SHHMSNV 
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When you have filled In ell of the blanks, reed the story again to see 
If It makes sense. When you have finished, look at the answers 
which ere written upside down at the bottom of the page. Change 
your wrong answers to the correct ones end reed the passage again! 

(Teacher reeds the paragraph Inserting the correct words) 

Pencils down. 

Now ere there any questions about how you take the test? #Durlng the • 
test I will not be able to answer any questions. 

All of these tests ere different. Some will take longer then others. 

After you finish close your test booklet, put It face down on the desk. 

Then Take out s book end reed. 

Please do not talk while the test Is In progress. 

You will have plenty of time. 

Open the booklet and fold the first page flat on the desk. 

You may begin 

(After 25 minutes sag STOP. .PENCILS DOWN --Ask the monitors to collect 
the tests and pencils.) 



#Call on children for a response. In some cases a response Is 
Indicated In parentheses. If It Is given say "yes" or "good". 
If the response In parentheses Is not given say "no" and call 
on another child. Repeat the question If necessary. 
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MULTIPLE-CHOICE TEST DIRECTIONS 



Good morning, my name la . 

Your teacher may have told you that I am going to give you a teat. 

It la Important for you to do the beat job you can, but, the acore will 
not be put on your achool record*. 

Pleaae clear your deaka. 

I need four volunteera. (direction* to volunteer*) 

Two of them paee out the teata to the pereon named ao that the 
name la up. 

TWo of them hand out the pencil*. 

Pleaae do not turn the booklet over until I ask you to do ao. 

If your pencil break* during the teat, ralae your hand and I'll 
give you another. 

Pleaae leave the pencil on the deak until we a tart the teat. 

(When all of the material* have been handed out, SAY ) 

NOW turn over the teat booklet . Find the word DIRECTIONS . 

Look at Sample (A) Read the atory silently while I read It aloud. 

About two hundred years ago, the first big balloon waa made. 

The man who made the balloon wanted some animals to try It out. 

He put a sheep, a rooster, and a duck In the balloon and sent 
the balloon Into the air. Soon It came down. The balloon ride 
had not hurt the animals. 

After each atory there are unfinished sentences about the story and three 
or four possible endings. Your job Is to decide which of the endings Is 
best and put a circle around the letter In front of that ending. 

Sentence number one says The first balloon was made 

A. last year. 

B. 200 years ago. 

C. 2000 years ago. 

D. 20 years ago. 



24 



Put a circle around the letter in front of the ending that you think ia beat. 
You nay look back at the story if you need to. 

Which letter did you circle? #(B) 

Now read the second sentence and circle the letter in front of the beat ending. 
Which letter did you circle? #(A) 

Now. ..Put your pencils down. 

Are there any questions about how you take thia teat?# 

All of these tests are different. Some of them will take longer than othera. 
When you finish one story, go on to the next. 

Be sure that you complete all of the sentences on each page. 

You will have plenty of time. When you finish you may go back and check 
your answers. 

When you feel that you have done the best you can, close the booklets and 
put them face down on the desk. 

Then... Take out a book and read. . .Please do not talk . 

Now. ..Are there any questions?# 

Open the first page, fold it flat on the desk. 

You may begin. 

(At the end of 25 minutes say STOP- -PENCILS DOWN- -Ask the monitors 
to collect the tests and the pencils.) 

#Call on children for a response. In some cases a response is indicated 
in parentheses. If it is given say "yes" or "good". If the response 
in parentheses is not given say "no" and call on another child. Repeat 
the question if necessary. 



